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Chapter 4 Symmetric matrices and the second 
derivative test 


In this chapter we are going to finish our description of the nature of nondegenerate critical 
points. But first we need to discuss some fascinating and important features of square matrices. 


A. Eigenvalues and eigenvectors 


Suppose that A = (a;;) is a fixed n x n matrix. We are going to discuss linear equations 
of the form 





Ax = Xz, 


where x € R” and \ € R. (We sometimes will allow x € C” and A € C.) Of course, x = 0 
is always a solution of this equation, but not an interesting one. We say x is a nontrivial 
solution if it satisfies the equation and x # 0. 

















DEFINITION. If Av = Xx and x F 0, we say that A is an eigenvalue of A and that the 
vector x is an eigenvector of A corresponding to A. 


0 3 


EXAMPLE. Let A= (; 9 


) Then we notice that 


a(1)=() =): 


1\. : . . 
so (;) is an eigenvector corresponding to the eigenvalue 3. Also, 


3 —3 3 
4(2)-G)=-(), 
3 \. ‘ : ; 
so (2) is an eigenvector corresponding to the eigenvalue —1. 


EXAMPLE. Let A= ¢ a Then A (;) =—2 (;). so 2 is an eigenvalue, and A (2) = 


Gr so 0 is also an eigenvalue. 


REMARK. The German word for eigenvalue is eigenwert. A literal translation into English 
would be “characteristic value,” and this phrase appears in a few texts. The English word 
“eigenvalue” is clearly a sort of half translation, half transliteration, but this hybrid has stuck. 
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PROBLEM 41. — Show that A is invertible ==> 0 is not an eigenvalue of A. 


The equation Ax = Ax can be rewritten as Ax = AI x, and then as (A—AI)x = 0. In order 
that this equation have a nonzero x as a solution, Problem 3-52 shows that it is necessary 
and sufficient that 

det(A — AI) = 0. 


(Otherwise Cramer’s rule yields x = 0.) This equation is quite interesting. The quantity 


ay4— rv a42 bie Ain 
ag) aj2 — : eee Q2n 
det ; 
Ant An2 see) Ann — 


can in principle be written out in detail, and it is then seen that it is a polynomial in A of 
degree n. This polynomial is called the characteristic polynomial of A; perhaps it would be 
more consistent to call it the ezgenpolynomial, but no one seems to do this. 
The only term in the expansion of the determinant which contains n factors involving \ is 
the product 
(a1, — A)(@22 — A)... (G@nn — ). 


Thus the coefficient of \" in the characteristic polynomial is (—1)". In fact, that product is 
also the only term which contains as many as n — 1 factors involving , so the coefficient of 
A”* is (-1)""! (aq, +. G22 +++ +Gnn). This introduces us to an important number associated 
with the matrix A, called the trace of A: 


traceA = ayy + dogg +--+ + Ann. 


Notice also that the polynomial det(A — AJ) evaluated at \ = 0 is just det A, so this is the 
constant term of the characteristic polynomial. In summary, 


det(A — AF) = (—1)"X" + (—1)""1(traceA)A" | + --- + det A. 


PROBLEM 4-2. Prove that 


traceAB = traceBA. 
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EXAMPLE. All of the above virtually provides an algorithm for finding eigenvalues and 
eigenvectors. For example, suppose 
1 2 
As (; | 


We first calculate the characteristic polynomial, 


ee ee veer es ‘ ) 


1 8=j 
=(1—A)(3-A)-2 
= \? 4) 41. 


Now we use the quadratic formula to find the zeros of this polynomial, and obtain \ = 
2 + \/3. These two numbers are the eigenvalues of A. We find corresponding eigenvectors 7 
by considering (A — AJ)x = 0: 


CP eva) ) = (0): 


We can for instance simply choose a solution of the lower equation, say x1} = 14 V3, x) = —1. 
The upper equation requires no verification, as it must be automatically satisfied! (Neverthe- 
less, we calculate: (—1 + V3)(1 = V3) + 2(—1) = 2—2 = 0.) Thus we have eigenvectors as 


follows: 
0-9) - ea) 
a(t hy) (2 — V3) ey 


=] a! 
0 1 
so, 


The characteristic polynomial is \? + 1, so the eigenvalues are not real: they are +i, where 
i = /-1. The eigenvectors also are not real: 


a(t) = (41) = «(2): 
= 


EXAMPLE. Let 
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Of course, the moral of this example is that real matrices may have only nonreal eigenvalues 
and eigenvectors. (Notice that this matrix is not symmetric.) 


EXAMPLE. Let 
2. ob ot 
A= {0 2 1 
00 2 


The characteristic polynomial is clearly (2 — \)?, so \ = 2 is the only eigenvalue. To find an 
eigenvector, we need to solve (A — 2/)x = 0. That is, 


0 11 Uy 0 
00 1 2} = {0}, 
0 0 0 X3 0 
or equivalently, 
tg+%3 = 0, 
v3 = 0. 
C 
Thus the only choice for x is x = | 0]. Thus there is only one linearly independent eigen- 
0 


vector. 


PROBLEM 4-3. Modify the above example to produce a 3 x 3 real matrix B whose 
characteristic polynomial is also (2—.)°, but for which there are two linearly independent 
eigenvectors, but not three. 


Moral: when A is an eigenvalue which is repeated, in the sense that it is a multiple zero of 
the characteristic polynomial, there might not be as many linearly independent eigenvectors 
as the multiplicity of the zero. 


PROBLEM 4-4. Let Xo be a fixed scalar and define the matrix B to be B = A— ol. 
Prove that 4 is an eigenvalue of A <> A — Xo is an eigenvalue of B. What is the relation 
between the characteristic polynomials of A and B? 


PROBLEM 4-5. | If Ais ann x n matrix whose characteristic polynomial is \” and 
for which there are n linearly independent eigenvectors, show that A = 0. 
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EXAMPLE. From Problem 3-29, take 


A 


The characteristic polynomial of A is 


1-r -l 1 
det(A—AI)=det| -1 3-A 0 
1 0 2—A 
= (1 — A)(3— A)(2— A) — (8 - A) — (2--A) 
==)" 26) = 0) 1. 


The eigenvalue equation is 
dN? — 64? + 9\ —1 = 0; 


this cubic equation has three real roots, none of them easy to calculate. The moral here is 
that when n > 2, the eigenvalues of A may be difficult or impossible to calculate explicitly. 

Given any n X n matrix A with entries a,;; which are real numbers, or even complex 
numbers, the characteristic polynomial has at least one complex zero \. This is an immediate 
consequence of the so-called “fundamental theorem of algebra.” (This is proved in basic 
courses in complex analysis!) Thus A has at least one complex eigenvalue, and a corresponding 
eigenvector. 


PROBLEM 4-6. Calculate the eigenvalues and eigenvectors of the matrix 


ae ee | 
A= ]{-11 4 
1 2 =! 


PROBLEM 47. Learn how to use Matlab or Mathematica or some such program 
to find eigenvalues and eigenvectors of numerical matrices. 


Now reconsider the characteristic polynomial of A. It is a polynomial (—1)"\" +... of 
degree n. The fundamental theorem of algebra guarantees this polynomial has a zero — let us 
call it A,. The polynomial is thus divisible by the first order polynomial A — Aj, the quotient 
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being a polynomial of degree n — 1. By induction we quickly conclude that the characteristic 
polynomial can be completely factored: 


det(A — AI) = (—1)"(A— y)..- (A— An): 


We think of A,,...,A, as the eigenvalues of A, though some may be repeated. We can now 
read off two very interesting things. First, the constant term in the two sides of the above 
equation (which may be obtained by setting \ = 0) yields the marvelous fact that 





det A = A, 2 ads Xn: 











Second, look at the coefficient of \”~ in the two sides (see p. 4-2) to obtain 





traceA = Ay + Ag +--- +n. 











These two wonderful equations reveal rather profound qualities of det A and traceA. Although 
those numbers are explicitly computable in terms of algebraic operations on the entries of A, 
they are also intimately related to the more geometric ideas of eigenvalues and eigenvectors. 


B. Eigenvalues of symmetric matrices 


Now we come to the item we are most interested in. Remember, we are trying to understand 
Hessian matrices, and these are real symmetric matrices. For the record, 


DEFINITION. Ann x n matrix A = (a,;) is symmetric if aj; = aj; for all i, 7. In other 
words, if A’ = A. 

We have of course encountered these in the n = 2 case. The solution of Problem 3-18 
shows that the eigenvalues of the 2 x 2 matrix 


A B 

BC 
A+C+/(A-—C)? 4+ 4B? 
9 y) 


and these are both real. This latter fact is what we now generalize. 
If A is an n X n matrix which is real and symmetric, then Problem 2-83 gives us 


are 





vX = 








Arey=xeAy for all x,y € R”. 





PROBLEM 4-8. Prove conversely that if Arey = xe Ay for all x, y € R”, then A 
is symmetric. 
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THEOREM. /f A is a real symmetric matriz, then its eigenvalues are all real. 


PROOF. Suppose \ is a possibly complex eigenvalue of A, with corresponding eigenvector 
z €C”. Write \ and z in terms of their real and imaginary parts: 














A=a+iZ, where a, ER, 
z=x+iy, where x,y € R” and are not both 0. 





Then the eigenvalue equation Az = Az becomes 
A(x + iy) = (a +i8)(a + iy). 


That is, 
Ax +iAy = ax — By + i(ay + Ba). 


This complex equation is equivalent to the two real equations 


Ar =azxz— By, 
Ay =ayt+ Gx. 


We now compute 


Arey =arey—llyll, 
Ayer =arey + pllel?. 


Since A is symmetric, the two left sides are equal. Therefore, 


arey— Bllyl|’ =ax ey + Bllall’. 


That is, 
B(x |? + llyll?) = 0. 


Since |||? + ||y||? > 0, we conclude G = 0. Thus \ = a is real. 
QED 
We conclude that a real symmetric matrix has at least one eigenvalue, and this eigenvalue 
is real. This result is a combination of the profound fundamental theorem of algebra and the 
above calculation we have just given in the proof of the theorem. It would seem strange to 
call upon complex analysis (the fund. thm. of alg.) to be guaranteed that a complex root 
exists, and then prove it must be real after all. That is indeed strange, so we now present an 
independent proof of the existence of an eigenvalue of a real symmetric matrix; this proof will 
not rely on complex analysis at all. This proof depends on rather elementary calculus. 
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Even so, it may seem strange to rely on calculus at all, since we are trying to prove a 
theorem about algebra — roots of polynomial equations. However, simple reasoning shows 
that something must be used beyond just algebra. For we are using the real numbers, a 
complete ordered field. The completeness is essential, as for example the polynomial \? — 2 
illustrates. Or even more challenging, imagine an equation such as A!43—\+5 = 0; it definitely 
has a real solution. These two examples have only irrational solutions. 











Let A be the n x n real symmetric matrix, and consider the quotient function R” 4, R, 


Arex Axex 


%) = Ta = 





Lex 


This is a rather natural function to consider. In a sense it measures something like the relative 
distortion of angles caused by A. “Relative,” because the denominator ||z||? is just right for 
Q(x) to be scale invariant. Notice how geometry will be used in what follows to give our result 
in algebra — the existence of an eigenvalue. This function is known as the Rayleigh quotient. 

This function is defined and of class C® on R” — {0}, and we can compute its gradient 
quite easily. First, we have from Problem 2-84 a formula for the gradients of the numerator 
and the denominator: 





VArex = 2Az, 
V|la|/? = 2z. 


Thus the quotient rule yields 


||a:|?2Ax — (Are x)2x 








VQ(z) = 
||| 
2Ax 2Axex 
= — a. 
lr P [eel 
The function Q is continuous on the unit sphere ||z|| = 1. Since this sphere $(0,1) is closed 


and bounded, Q restricted to $(0,1) attains its maximum value. Say at a point 29, so that 
|zo|| = 1 and Q(x) < Q(ao) for all ||z|| = 1. But the homogeneity of Q shows that Q(z0) is 
also the global maximum value of Q on R" — {0}. (This argument probably reminds you of 
Problem 3-18.) The details: if c 4 0, then 2/||z|| is on S(0,1), so that 





x 


Q(x) =@ (=) < Q(e0). 


[2| 


Thus Zp is a critical point of Q (p. 2-36). That is, VQ(ao) = 0. Let \ = Az e xg. Then the 
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above expression for VQ gives 


2At% 22Axy) © Xp 
0 SS V = —_— 
Or) = Tag? ~~ Yecoll® 7° 


= 2Axo _ 2AXo- 





Therefore 
Ary = Axo, ||Zol| = 1. 


We conclude that A is an eigenvalue of A, and xp is a corresponding eigenvector! Moreover, 
this particular eigenvalue is given by 


A = max{Agres | |[x|| = 1}, 
and x9 is a point where this maximum value is attained! 


PROBLEM 4-9. Calculate the Hessian matrix of Q at a critical point x with 
||Zo|| = 1. Show that it is 


H = 2A-2M  (A=Q(z»)). 


The analysis we are going to do next will continue to use the quotient function and the 
formula we have obtained for its gradient, so we record here for later reference 





VQ(xz) = 2(Az—Q(e)z) for |\z|| =1. 











We are now doubly certain as to the existence of a real eigenvalue of the real symmetric 
matrix A. We proceed to a further examination of the eigenvector structure of A. First here 
is an incredibly important property with a ridiculously easy proof: 


THEOREM. Let x and y be eigenvectors of a real symmetric matrix, corresponding to dif- 
ferent eigenvalues. Then x and y are orthogonal. 


PROOF. We know that Ax = \yx and Ay = Agy and A; # A». Therefore 


Airey) = (Aiz)ey 
Arey 


xe Ay (because A is symmetric!) 
= re (Ay) 
= Ax(xey). 
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Subtract: 
(Ai —A2)rey = 0. 
Since A; — Ap #0, rey =0. 
QED 
Next we give a very similar fact, based on the identical reasoning. 


THEOREM. Assume A is ann x n real symmetric matrix. Assume x is an eigenvector of 
A, and let M be the ((n — 1)-dimensional) subspace of R" consisting of all points orthogonal 
to x: 








M = {ye R"|xey = 0}. 


Then M is tnvariant with respect to A. That is, 


ye M => Aye M. 


PROOF. So simple: if y € M, 
Ayex = yeAr = yedt = Ayer) = 0. 


Thus Ay € M. 
QED 
Looking ahead to Section D, we now see a very nice situation. We have essentially split 
IR” into a one-dimensional space and an (n — 1)-dimensional space, and on each of them 
the geometric action of multiplying by A is clear. For the one-dimensional space lies in the 
direction of an eigenvector of A, so that A times any vector there is just \ times the vector. 
On the (n — 1)-dimensional space M we don’t know what A does except that we know that 
multiplication of vectors in M by A produces vectors that are still in M. This situation 
effectively reduces our analysis of A by one dimension. Then we can proceed by induction 
until we have produced n linearly independent eigenvectors. 





C. Two-dimensional pause 


We are now quite amply prepared to finish our analysis of the structure of real symmet- 
ric matrices. However, I would like to spend a little time discussing a standard “analytic” 
geometry problem, but viewed with eigenvector eyes. Here is an example of this sort of 


PROBLEM. Sketch the curve in the x — y plane given as the level set 


102? = 122y + 5y? = 1. 
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10-6 
a= (2 3): 


and the curve is given in vector notation as 


a6) - 
y y 
Now we find the eigenvalues of A: 
10-—A -6 
det ( 6 5 \) 


so the eigenvalues are 1 and 14. The eigenvector for \ = 14 is given by solving 


aun (2) =) 
(96) - 0) 
S 


a S 

m= 75 (So) 
For the other eigenvalue 1 we can use a shortcut, as we know from Section B it must be 
orthogonal to ~,. Thus we let 


The associated symmetric matrix is 


Be se ee ome 


(A — 1)(A — 14), 


Thus we may use the vector 


Normalize it and call it Qi: 


n= taC) 


and we are guaranteed this is an eigenvector! (Here’s verification: 


tm = Fa(Zs 5) (9 


si 


| 
© 
iw) 
ie 
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Now we use the unit vectors ¢, and @2 as new coordinate directions, and call the coordi- 
nates s and t, respectively: 


‘) = sg, +t¢ 
y P1 2. 


We calculate: 


(7) ; (;) = (sAgy +tAgs) © (51 + tr) 


= (14s, + tho) e (shi + to) 
= 1457+ #?. 


(Notice: no term with st!) Thus we recognize our curve in this new coordinate system as the 
ellipse 


ide +P = 1, 


Now the sketch is easily finished: we simply locate ~, and @, and the rest is easy. Here is 
the result: 
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be 








What has happened here is clear. This ellipse is not well situated in the x — y coordinate 
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system. In other words, the directions €; and é€2 are not of much geometrical interest for 
it. But the directions ~, and @2 are extremely significant for this ellipse! In fact, ¢, is the 
direction of its minor axis, (2 of its major axis. We say that ~, and @ are the “principal 
axes” for the ellipse and for the matrix A. Notice of course that 1 and 2 are orthogonal. 
(Another way of expressing this achievement is to think of the bilinear form 10x? — 127y + 


5y? as the square of a certain norm of the vector (;). This is definitely not the Euclidean 


norm, of course. But it has essentially all the same properties, and in fact in the new coordi- 
nates s’ = /14s and t’ = t we have 


Gl=az s~itt¢ 
wa 


10a? — 122y + 5y? = (s')? ++ @), 
so that the ellipse looks like the unit circle in the new coordinates.) 
In the next section we are going to extend all of that to the n-dimensional case, and the 
result will be called the principal axis theorem. 
Here are some exercises for you to try. 


and 


PROBLEM 4-10. Carry out the same procedure and thus accurately sketch the 
curve in the x — y plane given by the level set 16x? + 4xy + 19y? = 300. 


PROBLEM 4-11. Repeat the preceding problem for the curve 23x? —72ry+2y? = 50. 


PROBLEM 4-12. A further wrinkle in problems of the sort just presented is the 
presence of first order terms in the equation. Here is the n-dimensional case. Let A be 
an n X n real symmetric matrix and c € R” and a € R and consider the set described by 

















Axex+cex = a. 
Suppose det A 4 0. Then reduce this situation to one of the form 
Ayey = b 


by a judicious choice of x in the translation x = 2 + y. This is called “completing the 
square.” The point is that in the x coordinates the center of the figure is x9, but in the 
y coordinates it is 0. 
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PROBLEM 4-13. = Accurately sketch the curve in the x — y plane given as the level 
set (x — 2y)? + 5y = 0. Show that it is a parabola, and calculate its vertex. 


D. The principal axis theorem 


Now we come to the result we have been eagerly anticipating. This result is of major 
importance in a wide variety of applications in mathematics, physics, engineering, etc. In our 
case it is definitive in understanding the Hessian matrix at a nondegenerate critical point. It 
has a variety of names, including “The Spectral Theorem” and “Diagonalization of Symmetric 
Matrices.” There is an important term used in the statement which we now define. 





DEFINITION. If yi, Yo,..., yx are vectors in R” which are mutually orthogonal and which 
have norms equal to 1, they are said to be orthonormal. In terms of the Kronecker symbol, 


Pi © Yj = Oij- 


Since the vectors have unit norm, we distinguish them with our usual symbol for unit vectors, 
Pi. 


PROBLEM 4-14. Prove that the vectors in an orthonormal set are linearly indepen- 
dent. 
(HINT: if eer ci~i = 0, compute the inner product of both sides of the equation with 


Q;-) 











Therefore it follows that if we have n orthonormal vectors in R” (same n), they must form 
a basis for IR”. See p. 3-37. We then say that they form an orthonormal basis. The coordinate 
vectors €1, €2,...,@, are a standard example. 
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PROBLEM 415. Here is an orthonormal basis for R?: 


vali)’ val): 


Similarly, find an orthonormal basis for R* for which each vector has the form 








= ll 
= ol 
= ll 
= all 


— 





Find an analogous orthonormal basis for R°. 





We found an orthonormal basis for R? in our ellipse problem at the end of Section C, 
namely 


a de(2) oF) 





PROBLEM 416. Suppose $1, 2,---,Pn are an orthonormal basis for R”. Prove 
that every x in R” has the representation 


n 
f= y Le Pi Vij. 
i=l 





Notice how very much the formula for x in the problem resembles the formula 


Ly 


n 
i=1 


In 


In fact, it is the generalization to an arbitrary orthonormal basis instead of the basis of 
coordinate vectors. 
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PROBLEM 4-17.  Orthonormal bases often provide nice information about determi- 
nants. Suppose ~}, ~2,.--, Yn are an orthonormal basis for R”, written as column vectors. 
Define the n x n matrix having them as columns: 














a. Prove that ®'@ = J. 

b. Prove that det ® = +1. 

c. Suppose A is a matrix such that the ¢,’s are eigenvectors: 
AG, = Ai¥i- 


Prove that 
A® = (A1¢1 see AnYn)- 


d. Prove that 
det A = Ay AQ sare An: 


PRINCIPAL AXIS THEOREM. Let A be ann x n real symmetric matrix. Then there 
exist eigenvectors (1, 2,---, Pn for A which form an orthonormal basis: 


AG; = Aes Leos me 


The eigenvalues 1, A2,..-,An are real numbers and are the zeros of the characteristic poly- 
nomial of A, repeated according to multiplicity. 


PROOF. We are confident about using the quotient function Q(z) = Axe x/|\x\|?._ We 
have already proved in Section B that an eigenvector ~, exists, and we are going to carry 
out a proof by induction on k, presuming we know an orthonormal sequence ¢,...,~, of 
eigenvectors. We assume 1 < k <n-—1. We define 





M = {yER"|yeQi=---=yed, = Of. 


(This is a subspace of R” of dimension n — k.) We restrict the continuous function Q to the 
closed bounded set MM S(0,1). It attains a maximum value there, say at a point 7. Thus 
||| = 1 and pj eG; =--- = Zp eG, = 0. Because Q is homogeneous of degree 0, we know in 
fact that Q(x) < Q(x) for all x € M; this is the same argument we used on p. 4-8. 
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This implies that for all h € M we have 
Q(fo + th) < Q(%o), —00o <t<o. 


And this gives a maximum value at t = 0, so that 


ae 
pe (eo +th) |,_, = 0. 


That is, the directional derivative DQ(io;h) = 0. That is, 
VQ(i0) eh = 0 for all he M. 


Now the boxed formula on p. 4~9 asserts that 

VQ(Ho) = 2 (Ax — Q(%o) 40) . 
We know from the theorem on p. 4-10 that Av € M, and thus VQ(%o) € M. But since 
VQ(o) is orthogonal to all vectors in M, it is orthogonal to itself, and we conclude VQ(io) = 
0. Thus Zp is a critical point for Q! 

That does it, for At = Q(%o)%o. We just name %p = pri and Q(%) = Axg41. We have 
thus produced an orthonormal sequence (j,..., x41 of eigenvectors of A. By induction, the 
proof is over, except for one small matter. That is the statement about the characteristic 
polynomial. But notice that 

(A-ANG: = Ai —A) Gi, 
and thus Problem 4—17 yields 
det(A — AT) = (A, —A).-- (An — A). 
QED 


REMARK. This is an unusual sort of induction argument. If you examine it carefully, you 
will notice that it really applies even in the case k = 0. There it is exactly the proof we gave in 
Section B. Thus we don’t even actually need the proof of Section B, nor do we need a separate 
argument to “start” the induction. This is quite a happy situation: the starting point of the 
induction argument is not only easy, it is actually vacuous (there’s nothing to check). 


PROBLEM 4-18. This is sort of an easy “converse” of the principal axis theorem. 
Given any orthonormal sequence %1,...,,, in R” and any real numbers \j,..., An, there 
exists one and only one n x n real matrix A such that 














Ag; = ri for all 1<i<n. 


Prove that A is symmetric. 
(HINT: use ® from Problem 4-17.) 
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PROBLEM 4-19. Find a4 x 4 matrix A such that 


1 0 | 2 
1 0 il 2 
A il (eee (em a —1 —2]? 
1 0 —1 —2 
1 —1 0 0 
—1 1 0 0 
A 0 = 0 |: and A l = 5 
0 0 —1 —5 


The sort of matrix that was introduced in Problem 4-17 is exceedingly important in under- 
standing both the algebra and the geometry of our Euclidean space R”. We need to understand 
all of this in great detail, so we pause to give the definition. 





DEFINITION. A real n x n matrix ® is an orthogonal matrix if its columns are an or- 
thonormal basis for R". That is, 





and ~; ¢ ~; = 6;;. The set of all orthogonal n x n matrices is denoted 


O(n). 


You noticed in Problem 4-17 that an equivalent way of asserting that ® is orthogonal is 
the matrix formula 6°@ = J. Thus, that ®° is a left inverse of ®. But the theorem on p. 3-37 
then asserts that ® is invertible and has the inverse ®'. Thus, ®&* = J as well. Here is a 
problem that summarizes this information, and more: 
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PROBLEM 4-20. Prove the following properties of O(n): 











a. ® € O(n) => the columns of ® form an orthonormal basis for R” (this is actually 
our definition). 





b. ® € O(n) <> © is the inverse of ©&. 





c BEO(n 








<=> the rows of ® form an orthonormal basis for R”. 

















e. BE O(n) => Ore by = vey for all x, y € R”. 





(n) 
(n) 
d. 6 € O(n) = F € O(n). 
(n) 
(n) 
( 








f. BE O(n) => ||Gz|| = ||z|| for all z € R”. 





g. BE O(n) = O 1 € O(n). 
h. ®, ® € O(n) = 60 € O(n). 


(HINT for f: the hard part is <=. Try showing that the condition in part e is satisfied, 
by verifying 
2bx e dy = || P(x + y)|/’ — ||Gx|? — ||Gyl|?.) 


DISCUSSION. Because of the last two properties in the problem, O(n) is called the or- 
thogonal group. The word “group” is a technical one which signifies the fact that products of 
group elements belong to the group, that there is an identity for the product (in this case it’s 
the identity matrix J), and that each member of the group has a unique inverse (which also 
belongs to the group). 

Notice how easy it is to compute the inverse of an orthogonal matrix! 


DEFINITION. The set of all n x n invertible real matrices is called the general linear group 
and is denoted 


GL(n). 


The set of all n x n real matrices with determinant 1 is called the special linear group and is 
denoted 


SL(n). 
Every orthogonal matrix has determinant equal to +1 (Problem 4-17). The set of all orthog- 
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onal matrices with determinant 1 is called the special orthogonal group and is denoted 
SO(n). 


Clearly, 
SO(n) C SL(n) C GL(n) 
and 


SO(n) = O(n) NSL(n). 


PROBLEM 4-21. Prove that GL(n), SL(n), and SO(n) are all groups. 











PROBLEM 4-22. Let (j,...,¢%, be an orthonormal basis for R", and A any n x n 
real or complex matrix. Prove that 





traceA = » Ai © Qi. 


i=1 


E. Positive definite matrices 


In this section we lay the foundation for understanding the Hessian matrices we are so 
interested in. 

Let A be an n x n real symmetric matrix. The principal axis theorem guarantees the 
existence of an orthonormal basis for R” consisting of eigenvectors of A: 





As we have discussed, the unit vectors ¢1,...,, are very natural as far as the matrix A is 
concerned. We now use them essentially as a new set of “coordinate axes” for R”. That is, 
every x € R” has a unique representation of the form 


n 
C= ) SiQi- 
i=l 


The numbers s1,..., 5, are the “coordinates” of x in this new basis. They can be calculated 
directly by using the inner product: 








5, = Le Yj. 
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Now we calculate the quadratic form we are interested in. In the Cartesian coordinates it 
is of course 


n 
Arexr= y ia. 
ij=l 


In the more natural coordinates it is computed as follows: 


Arex= se Ss; AP; e x 85D; 
i=l j=l 

= yi SiAiPi © ‘3 85Dj 
i=l =I 


»; AiSi8 5 Pi ° Pj 


ij=l 


3 Aj 848505; 


i,j=l 
n 
) 2 
i=1 


Of course, the orthonormality was of crucial importance in that calculation. An example 
of this sort of calculation appears in Section 4C. 

The result is that Avex looks much nicer in the coordinates that come from the eigenvectors 
of A than in the original Cartesian coordinates. We reiterate, 


n 
Arex = y dist. 
i=1 


In this form we can deduce everything we need to know about the quadratic form Ax ez. For 
instance, we know in case A is the Hessian matrix of a function at a critical point, then the 
critical point is a local minimum for the function if Av ex > 0 for all « 4 0. We see instantly 
that this condition is equivalent to A; > 0 for all 2: 


THEOREM. In the above situation the eigenvalues of A are all positive — > Axex > 0 for 
all ¢ € R” — {0}. 














PROOF. For the direction <=, apply the given inequality to « = ¢;. Then 0 < Ag; e ¢; = 
Ai; @ P; = A;. Thus all the eigenvalues of A are positive. This much of the theorem did not 


Symmetric matrices and the second derivative test 23 


require A to be symmetric. However, the converse direction = > relies on the principal axis 
theorem. According to the calculation given above, Ar ex = )> \;s? > 0 since all A; > 0, and 
Ax ex = 0 implies each s; = 0 and thus x7 = yy sip; = 0. 

QED 


DEFINITION. The real symmetric matrix A is said to be positive definite in case the above 
equivalent conditions are satisfied. That is, all the eigenvalues of A are positive. Equivalently, 
Azex > 0 for all x € R" — {0}. 

Of course, we say A is negative definite if all the eigenvalues of A are negative; equivalently, 
Axex <0 for all x 4 0; equivalently, —A is positive definite. 





PROBLEM 4—23. — Give an example of a real 2 x 2 matrix for which both eigenvalues 
are positive numbers but which does not satisfy Ax ex > 0 for all « € R?. (Of course, 
this matrix cannot be symmetric.) 














It is quite an interesting phenomenon that positive definite matrices are analogous to 
positive numbers. The next result provides one of the similarities. 


THEOREM. A real symmetric matrix A is positive definite <> there exists a real symmetric 
invertible matriz B such that A = B?. 


PROOF. If A= B?, then Arex = B’xrex = Bre Br = ||Bx\|? > 0, and equality holds 
<=> Br = 0 <> « = 0 (since B is invertible). Conversely, use the eigenvectors ~; of A to 
define the orthogonal matrix 








® = (G1 G2 .-- Gn) 
Then 
Pi 
gi, 
M1 0 
— 2 ‘ 
0 Ay 
so that 
At 0 
r 
A= : ot 
0 
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Now simply define 
VA 0 
\ 
B= Va o!, 
O 7: 
Vn 


Then B is even positive definite, and B? = A. (We say B is a positive definite square root of 
A.) 
QED 
What has happened in the above proof is tremendously interesting, probably more in- 
teresting than the theorem itself. Namely, starting with A we have used the principal axis 
theorem to represent it as simply as possible in coordinates tied closely to the geometry which 
A gives. In that coordinate system it is easy to find a square root of A, and then we “undo” 
the coordinate change to get the matrix B. 


PROBLEM 4-24. _ Find a positive definite square root of 


16 2 
A=(5 o 


(see Problem 4-10). 


PROBLEM 4-25*. Prove that a positive definite matrix A has a unique positive 
definite square root. (For this reason, we can denote it V/A.) 

(HINT: suppose B is positive definite and B? = A. Show that if \ is an eigenvalue of A 
and Ar = Ag, then Ba = Vx.) 


PROBLEM 4-26. — Show that (; s) has no square root whatsoever. That is, there 


is no 2 x 2 matrix B even with complex entries such that B? equals the given matrix. 


PROBLEM 4-27. Prove that if A is positive definite, then so is A7?. 


Now we are ready to focus our attention on the real issue we want to understand. Re- 
member, we are trying to understand how to detect the nature of critical points of real-valued 
functions. Referring to Section 3H, “Recapitulation,” we see that the crucial quantity is Hyey, 
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where H is the Hessian matrix at the critical point of the function. We assume the critical 
point is nondegenerate, in other words that det H # 0. Now think about whether we have a 
relative minimum. This translates to Hy ey > 0 for all y 4 0, as we shall prove in Section F. 
In other words, the condition for a relative minimum is going to be that H is positive definite. 

Thus we are facing an algebra question: how can we tell whether a symmetric matrix 
is positive definite? The immediate but naive answer is just to respond: precisely when its 
eigenvalues are all positive. 

However, we know that this is a potentially difficult matter for n x n matrices with n > 2, 
as calculating the eigenvalues may be difficult. In fact, usually only numerical approximations 
are available. The truly amazing thing is that there is an algorithm for detecting that all 
the eigenvalues of a symmetric matrix A are all positive, without actually calculating the 
eigenvalues of A at all. The fact is, we have observed this very feature in the n = 2 case. For 
we know (Problem 3-18) that A is positive definite <> 


ay, > 0, ag2 > 0, and a4 1a22 — ee > 0. 
In fact, we could drop the second inequality and simply write 
a4, >0 and detA>0. 


Notice that calculating the eigenvalues in this case requires the square root of (a,;—4a2)?+4a7,, 
but our test requires no such thing. 
The n x n case has a similar simplification: 


THE DEFINITENESS CRITERION. Let A be ann xn real symmetric matrix. For any 
1<k<n, let A(k) be thek xk “northwest” square submatrix of A: 


ai Qik 
A(k) = 
Qk1 Qkk 
(Thus, 
A(1) = (ai), 
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Then A is positive definite <> det A(k) > 0 for alll <k <n. 
(By the way, we could just as well have elected to employ the corresponding southeast 
submatrices instead. More about this after the theorem.) 


PROOF. We first make a simple observation: if a matrix is positive definite, then its 
determinant is positive. The reason is that its determinant is equal to the product of its 
eigenvalues (p. 4-6), which are all positive. 

It is rather evident that the direction = > of the proof should be the easier one, so we 
attack it first. Suppose that A is positive definite. Then we prove directly that each A(k) 
is positive definite; the above observation then completes this part of the proof. For a fixed 
1<k<vn, let y € R¥ be arbitrary, y 4 0. Then define x € R” by 








Then it is true that 
Arex = A(k)yey. 


Since A is positive definite, Arex > 0. Thus A(k)yey > 0. Thus A(k) is positive definite. 

Now we prove the converse direction <=. We do it by induction on n, the case n = 1 being 
obvious. Thus we assume the result is valid for the case n — 1, where n > 2, and we prove it 
for an n X n matrix A. Thus we are assuming that each A(k) has positive determinant. By 
the induction hypothesis, A(n — 1) is positive definite. 

We now use the principal axis theorem to produce orthonormal eigenvectors for A. Ac- 
tually, for the present proof it is convenient to assume only that they are orthogonal (and 
nonzero), and that all of them with n‘* coordinate nonzero have been rescaled to have n‘® 
coordinate equal to 1: 


1,---;Yn are orthogonal and nonzero; 


the n™ coordinate of each y; is 0 or 1. 


By Problem 4-17, 
O < detA = A; Ao .-- An- 
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Each y; with n“ coordinate 0 can be written 





where y € R™! and y # 0, so that 


Ailleall? = Api e Yi 


= Api © Yi 
= A(n—l)yey 
> 0, 


since A(n — 1) is positive definite. Thus A; > 0. 

Now suppose two of the eigenvectors y; and y; have n‘ coordinate 1. Then y; — y; has 
coordinate 0 and is not itself 0, so as above we conclude that since A(n — 1) is positive 
definite, 


pth 


>) 
/\ 


A(yi — 3) © (Yi — Y3) 
= (Aipi — Aj¥;) © (Pi — 3) 
Aill vill? - A; llesll? (by orthogonality). 


Thus at least one of A; and A, is positive. 

This leads to an interesting conclusion indeed! Among all the eigenvalues \;, A2,..-, An; 
at most one of them is negative! Since their product is positive, they must all be positive. 
Therefore, A is positive definite. 

QED 
DISCUSSION. For any n x n real symmetric matrix A with det A 4 0, we now completely 
understand the criterion for A to be positive definite. Next, A is negative definite <> —A 
is positive definite <> det(—A(k)) > 0 for all k = (—1)* det A(k) > 0 for all k. Thus we 
obtain the negative definite result for free. 


SUMMARY. When we examine the signs of the determinants det A(k) in order for k = 1, 
2,...,n, there are exactly three cases: 


e +,+,+,+,...< > A is positive definite. 


e -,+,-,+,... <> Ais negative definite. 
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e any other pattern ==> A is neither positive nor negative definite. 


PROBLEM 4-28. State and prove the corresponding criterion, using instead the 
k x k southeast square submatrices of A. 


PROBLEM 4-29. Let A be an n x n real diagonal matrix with det A 4 0. Show 
that the definiteness criterion is virtually obvious for A. (Thus the useful content of the 
criterion is for matrices which are not diagonal.) 


THE DEGENERATE CASE. The definiteness criterion of course deals only with the 
nondegenerate case in which det A # 0. There is a companion result which is valid even if 
det A = 0. Although this criterion appears to be of little interest in the classification of critical 
points, since we need them to be nondegenerate, we include the material in the rest of this 
section for the beautiful mathematics that is involved. We continue to work with an n x n 
real symmetric matrix. Such a matrix A is said to be positive semidefinite if 





Axex > 0 forall 2 € R”. 


Equivalently, all the eigenvalues of A are nonnegative numbers. 
What you might expect the definiteness criterion to assert is that the equivalent condition 
is det A(k) > 0 for all 1 < k < n. However, a simple 2 x 2 example belies this: 


(A). 


The key is to realize that our restriction to northwest square submatrices is rather artificial. 
Instead we should use all possible “symmetric square submatrices.” These are matrices A’ 
obtained from A by using only the entries a;; where 7, 7 are restricted to the same collection 
of indices. Put in negative terms, we have deleted some rows of A as well as the corresponding 
columns. Whereas there are n symmetric square submatrices of the form A;, there are 2” — 1 
symmetric square submatrices in all. 


THE DEFINITENESS CRITERION BIS. Let A be ann x n real symmetric matriz. 


Then A is positive semidefinite <=> every symmetric square submatrix A’ satisfies 
det A’ > 0. 
PROOF. The => direction of the proof is just as before. The <= direction is again proved 


by induction on the dimension; the n = 1 case is trivial and we presume the n — 1 case is 
valid. 
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We again use a principal axis decomposition as before, 


Vis ereg Gy are orthonormal. 


If det A 4 0, then the theorem is already known from the previous result, so there is nothing 
to prove. Thus we may assume det A = 0, so that A has 0 as one of its eigenvalues. We may 
as well assume A; = 0. Now consider any 2 <7 <n. There exists a scalar c such that ~; — cp 
has at least one coordinate equal to 0 (it could happen that c = 0). Say its j** coordinate is 
0. Then we choose the particular symmetric square submatrix A’ obtained by deleting the j'® 
row and the j column from A (thus A’ equals the minor Aj; as defined on p. 3-29). Also 
let y € R"' be obtained from y; — cy, by simply deleting its j** (zero) entry. 
By the inductive hypothesis, A’ is positive semidefinite. Therefore 














0 < Alyey 
= A(¥; — chi) © (Pi — cf1) 
= (Ag; — cA) © (Gi — cP) 
(Afi — CA1¢1) © (Pi — CP1) 
APie (~i-— chi) (Ar = 9) 
NiPi © Pi ((d; and ¢, are orthogonal) 
= Aj. 


Thus A; > 0 for all 2<i<n (and ; = 0). Thus A is positive semidefinite. 
QED 


PROBLEM 4-30. Assume A is positive semidefinite and a;; = 0. Prove that the 
i+ row and the i" column of A consist of zeros. Prove that if A is positive definite, then 
ai; > 0. 


PROBLEM 4-31. Suppose A is an n x n positive definite matrix. Prove that 


traceA 





(det A)n < 


and that equality holds — > A =clI for some c > 0. 
(HINT: use the arithmetic-geometric mean inequality (Problem 5-31) for the eigenvalues 
of A.) 
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PROBLEM 4-32. Suppose A is an n x n positive definite matrix. Prove that 
det A < Qi, 492 ... Ann- 


Prove that equality holds == A is a diagonal matrix. 

(HINT: let B = the diagonal matrix with entries ,/aj. Let C = B~!AB™ and apply the 
preceding problem to C.) 

PROBLEM 4-33. Suppose A is an n x n positive semidefinite matrix. Prove that 


det A < aq da... Ann; 


and that equality holds <=> A is a diagonal matrix or some a;; = 0. 














PROBLEM 4-34. Suppose A is an n x n real matrix with columns ay,...,a@, in R”: 
A = (a1 a2 ... Gn). 


Show that Problem 4-33 may be applied to the matrix B = A‘A, and results in what is 
known as Hadamard’s inequality: 


|det A] < |lay|| |la2l] ..- [lanl 
When can equality hold? 


We shall see in Section 8A that Hadamard’s inequality has a very appealing geometric 
interpretation: the volume of an n-dimensional parallelogram is no greater than the product 
of its edge lengths. 

There is another approach to the analysis of positive semidefinite matrices that is quite 
elegant. This approach is completely algebraic in nature and thus entirely different from that 
we have seen thus far. It begins with a discussion of the determinant of a sum of two matrices. 
Suppose then that A and B are matrices represented in terms of column vectors in the usual 
way: 














Thus a; and b; € R”. Then the multilinearity of the determinant represents det(A+ B) as 
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a sum of 2” determinants, where each summand is the determinant of a matrix C' of the form 
(cy C2) eae Cas 


where each c; is either a; or b;. 
(This is very similar to the binomial expansion of (x + y)” when the power is regarded as 


(eg) = (ety) Gay). Gg) 


and all the multiplications are carried out, resulting in 2” terms.) 
Now specialize this formula to the case B = AJ. Then b; = Aé;, and when n — k of the 
columns of C' come from the b,’s, the resulting determinant is 


det C = X"-* det A’, 
where A’ is the k x k square submatrix of A resulting from eliminating the particular n — k 


rows and columns of A corresponding to this choice of C. Thus 


n 


det(A+A21)=Soa"* SYS” det A’, (*) 


k=0 A’ is kxk 


k 
deleting k rows and the same & columns. For instance, the n = 2 case is 


where each A’ in the inner sum is one of the (;,) square submatrices of A resulting from 


det (A + AT) = ae + (a1 + 22) + det A. 


(Notice that when k = 0 we are required to interpret the coefficient of \” as 1.) 

Notice that replacing 4 by —A in (*) gives an explicit formula for the characteristic poly- 
nomial of A. 

Here then is the punch line. Suppose we want to prove the hard direction <= of the 
definiteness criterion for positive semidefiniteness. We thus assume A is symmetric and every 
A’ satisfies det A’ > 0. Then for all \ > 0 we have det(A + AI) > 0. Therefore, if is an 
eigenvalue of A, det(A — AJ) = 0 and we conclude that —\ < 0. Thus all the eigenvalues of 
A are nonnegative, proving that A is positive semidefinite. 


PROBLEM 4-35. Prove that a positive semidefinite matrix has a unique positive 
semidefinite square root. 


F. The second derivative test 
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We return at last to the calculus problem we were interested in, as summarized at the close 
of Chapter 3. We use that outline exactly as written, and we assume that the critical point 
Xo for the function f is nondegenerate, so that the determinant of the Hessian matrix H is not 
zero. The test we are going to state is in terms of the definiteness of H, and we realize that 
the definiteness criterion of Section E may be useful in deciding this in a given case. However, 
we do not need to refer to the rule in the statement of the result. 


THEOREM. Assume the following: 











© R 4SRis of class C? in a neighborhood of xo. 
© 29 is a critical point of xo. 

© Xp is nondegenerate. 

e 6A is the Hessian matrix of f at xo. 

Then the conclusion is definitive: 


e =f has a strict local minimum at x) <=> H is positive definite. 
e = f_ has a strict local maximum at x9) <> H is negative definite. 
e =6f has a saddle point at x9 <= > H is neither positive nor negative definite. 


PROOF. We have the Taylor expansion from Section 3B, 


f(tot+y) = f(to) + sHyey +R, 


where |R| is smaller than quadratic as y — 0. 
e Assume H is positive definite. Then we use a principal axis representation for H as on 


p. 4-22, writing 
y= > SiPi, 
i=1 


so that 
Hyey = > ris? 
i=1 


All \; > 0, so let A = min(Ay,...,An). Then A > 0 and 


Hyey > AY 87 = Allyl?. 


i=1 
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Choose 5 > 0 such that for ||y|| <6 we have |R| < 4||y||?. Then 0 < ||y|] <6 => 


IV 


flvo+y) > Feo) + >All? — [RI 


1 2 if 2 
F (wo) + 5Allyl? — ZAlyll 


(0) + Flv? 


Thus f has a strict local minimun at 29. 

e If H is negative definite, the same proof yields a strict local maximum at Xo (or simply 
apply the previous result to —f). 

e If H is neither positive nor negative definite, then since all its eigenvalues are nonzero, it 
must have a positive eigenvalue and a negative eigenvalue. Suppose for example that A; < 0. 
Then 


IV 


ae 
f(@0) + ghee 


= f(%o)+ aut? ae 


f (xo + tpi) 





1 
F(a) + 5 it? te | A 


IA 


Now choose 6 > 0 so that 
1 
ly] < 6 |R&)| < — 7 rillull? 
Then 0 < |t] <6 ==> 


1 1 


y, 
= Fao) + ga 


Thus f does not have a local minimum at x9. Likewise, using a positive eigenvalue shows that 
f does not have a local maximum at x9. Thus x9 is a saddle point. 

Thus far we have covered the three implications <=. But since the three assertions on the 
left sides of the statements as well as on the right sides are mutually exclusive, the proof is 
finished. 
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QED 


G. A little matrix calculus 


We take the viewpoint of Section 31, thinking of n x n real matrices as being the Euclidean 
space R”’. Now we want to think of the calculus of the real-valued function det. 


PROBLEM 4-36. Use the formula (*) of Section 4E to write 


det(A + AI) =A" +A" traceA +A" FR+..., 


k= > (a0; = Oigde 


1<i<j<n 


PROBLEM 4-37. In the preceding problem perform algebraic manipulations to 
rewrite 


S-(aiiag; — 445038) 


tJ 


= 5 [(traceA)? — trace(A’)]. 


PROBLEM 4-38. Manipulate Problem 4-35 in such a way to achieve the polynomial 
equation 
det(J + tB) =I+ttraceB + higher order terms in t. 


Conclude that the differential of det at J is the linear mapping trace. In terms of direc- 
tional derivatives, 
D det(I; B) = traceB. 
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PROBLEM 4-39. Generalize the preceding result to obtain 


D det(A; B) = trace(BadjA). 





